Automatically generated visual annotated graphic legend

ABSTRACT

A method for automatically generating a hardcopy photo product with a visual annotated graphic legend provides additional information related to the printed image without obscuring the printed image.

FIELD OF THE INVENTION

The present invention relates to methods for automatically generating ahardcopy photo product with a visual graphic legend to provideadditional information related to the printed image.

BACKGROUND OF THE INVENTION

With soft display images presented on interactive devices such aspersonal computers, touch screen tablets, and smart phones, individualname tags that identify the people present in the recorded scene can beprovided as an interactive layer. This interactive layer is only visiblewhen the option is selected by the user, or the user touches the screenarea relative to the individual of interest, or operates a pointingdevice to “hover over” the image of an individual. When this is done thename for the individual appears over or adjacent to the individual anddisappears when the user returns to the normal viewing mode. This softdisplay feature provides a way to identify the individuals in a groupscene without obscuring the individuals or scene elements with apermanent text overlay. The present invention provides thisfunctionality in hard copy print formats.

Creating “coloring books” and “paint by numbers” drawings by convertingconsumer generated photos into line drawings is known and demonstratedby U.S. 20080284791A1 FORMING COLORING BOOKS FROM DIGITAL IMAGES. Theline-art image is formatted to produce a coloring book image and thecoloring book image is printed. In addition, this publication disclosesthat an index number may be assigned to a corresponding sample color andthe index number and color may be printed with the coloring book imageto produce a color-by-numbers coloring book image. U.S. 20080284791A1does reference generating different types of content, such assilhouettes for unsupervised coloring, borders with numbered regions,and using a color photograph and the coloring book line drawing so thatthe combination image and line drawing comprises color photographicsections substituted for corresponding portions of the monochromaticline-art as part of the graphic user interface (GUI) to verify that theline image shapes and assigned colors are correct.

U.S. 20020003631 SYSTEM AND METHOD FOR PRODUCING A COLORING BOOK IMAGEFROM A DIGITAL IMAGE also describes techniques creating “coloring books”and “paint by numbers” drawings by converting consumer generated photosinto line drawings color samples are generated from the digital imageand a fixed or programmable palette of colors is assigned to image areasof the coloring book image. In addition, an index number is assigned toa corresponding sample color and the index number and color is printedwith the coloring book image to produce a color-by-numbers coloring bookimage. The coloring book or color-by-numbers coloring book images can becombined with stock line-art images to generate a storyboard which maybe printed in the form of a coloring book.

However, neither U.S. 20020003631 nor U.S. 20080284791, produces a linedefining the outline each individual with the proper configurationindicating the relative depth or row in which the individuals within ascene are placed, or provide a means to identify those individualswithout obscuring the original scene content on a printed hardcopymedia. Typically this sort of graphically annotated group picture isprofessionally produced by graphic artists employing a range ofspecialized skills, image manipulation tools, and can be very costly forthe customer. In addition, manually creating an annotated visualreference for use with a hardcopy scene image is a burdensome taskrequiring a diverse range of image editing and processing skills and isill-suited for applications involving hardcopy photo products createdwith the use of smart phones, touch tablets, personal computers, andretail photo kiosks. These types of approaches for creating custom photoproducts assume that the user has no prior knowledge of imagemanipulation techniques or has any of the required image editing skills,but has the desire to purchase a custom photo product that meets theirneeds, easily and affordably. In the case of a retail photo kiosk, theuser's time within the retail establishment for a “do-it-yourself” or“while-you-wait” service is inherently limited by the user's willingnessto linger in a retail establishment for extended periods of time and thetime available for working at the kiosk can be limited due to otherusers waiting in line for their turns at the device.

What is required is a means of indicating the identities of individualsin group scenes on hardcopy print media when viewed without obscuring,editing, or modifying elements of the original scene. An automaticallygenerated annotated visual scene derived reference for use inassociation with the group scene, provides the identity information anda visual reference relative to the original scene. The annotated visualreference may be printed on the same side of the same media and adjacentto the original scene, on the back side of the same media as theoriginal scene, in the same or difference scale, on a separateconventional or transparent media used in conjunction with the originalscene print, on a multiple image viewing print media incorporating abarrier or lenticular overlay, or on a duplex translucent media withscene image on one side and an aligned and reverse printed annotatedvisual scene derived reference on the other side of the translucentmedia.

SUMMARY OF THE INVENTION

According to the present invention, a method of creating a hardcopyphoto product of a recorded scene of a group of individuals includesobtaining an image of a scene with two or more individuals; using aprocessor to analyze the scene to automatically create a line graphicthat outlines the individuals within the scene while maintaining theirpositions relative to the scene recording device; using the processor toextract the created line graphic image from the scene image; using theprocessor to obtain identification of individuals within the scene;using the processor to annotate the created line graphic image with theobtained identification information; using the processor to format therecorded scene image and the annotated line graphic image to produce ahardcopy photo product that includes the annotated line graphic image toprovide visual identification information related to the individualsdepicted in the scene image without obscuring the scene image; andproducing the formatted hardcopy photo product. The identification ofthe individuals within the scene may be obtained from facialrecognition, metadata related to the recorded scene image, a useroperating a user input device, or information provided by a socialnetwork. According to the method of the present invention the two ormore individuals within the scene can include people and/or animals,with individual name designations. The method of the present inventioncan be performed using a phone, tablet, personal computer, or retailphoto kiosk either locally on the device or connected via acommunications network to a remote processor. According to the method ofthe present invention the formatting of the photo product includes thescene image of one side of a print media and the annotated line graphicimage on the other, the scene image and the annotated line graphic imageon separate portions of the same side of a print media, the scene imageprinted on reflective media and the annotated line graphic image printedon a separate transparent print media, the scene image and a reducedscale version of the annotated line graphic image on separate portionsof the same side of a print media, the scene image printed on reflectiveportion of a print media and the annotated line graphic image is reverseprinted on an adjacent and attached transparent print media, the sceneimage and the annotated line graphic image are interspersed and printedon a lenticular or barrier multi-view print media, or the scene imageprinted on front surface translucent or transparent print media and theannotated line graphic image is reverse printed and aligned to the sceneimage on the rear surface of the translucent or transparent print media.The format for the photo product is selected from a group of softcopydisplayed format options by the user. The analysis of the scene contentto automatically create a line graphic that outlines the individualswithin the scene maintaining their positions relative to the scenerecording device includes at least one algorithmic technique includinggraph cut image segmentation, edge detection, pose consistency, poseclustering, scale-invariant feature transform (SIFT), speeded up robustfeatures (SURF), and/or simple interactive object extraction (SIOX). Theprocessor can be used to perform all of the steps of the method of thepresent invention automatically.

In summary, the method of the present invention automatically generatesa hardcopy photo product with a visual annotated graphic legend toprovide additional information related to the printed image withoutobscuring the printed image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a graphic illustration showing an image processing systemsuitable for practicing the image processing method according to thepresent invention.

FIG. 2a is a flow chart showing one embodiment of the image processingmethod according to the present invention.

FIG. 2b is a flow chart showing the image process method for creatingthe outlines around the identified subjects.

FIG. 2c is a flow chart showing the image process method for determiningthe Z-order positions for the outlines around the identified subjects.

FIG. 3a is an illustration of a group scene showing six individuals.

FIG. 3b is an illustration of a group scene showing six individuals withlines formed around the edges of each individual.

FIG. 3c is an illustration of a group scene showing six individuals withlines formed around the edges of each individual and the interior areaof each individual removed.

FIG. 3d is an illustration of a group scene showing six individuals withlines formed around the edges of each individual, the interior area ofeach individual removed, and the background of the scene removed.

FIG. 4a is an illustration of a front surface of a duplex printdepicting a group scene showing six individuals according to the presentinvention.

FIG. 4b is an illustration of a rear surface of a duplex print depictinga group scene showing six individuals in outline with annotationsprovided for each individual to provide information relevant to thefront surface of the print according to the present invention.

FIG. 5 is an illustration of the front surface of a print depicting agroup scene showing six individuals in one area, an illustrationdepicting the same group scene showing six individuals in outline eachwith an alpha numeric indicator in a second area, and a legend providingidentification information relative to the alpha numeric indicators in athird area.

FIG. 6a is an illustration of a conventional print of a group sceneshowing six individuals.

FIG. 6b is an illustration of a transparency print depicting the samegroup scene showing six individuals in outline each with identificationlabel within the outlines of each individual.

FIG. 6c is an illustration of a transparency print aligned andoverlaying the conventional print of a group scene showing sixindividuals

FIG. 7 is an illustration of a conventional print of a group sceneshowing six individuals with an adjacent annotated outline printed onthe same media.

FIG. 8 is an illustration of a conventional print of a group sceneshowing six individuals with an alternative adjacent annotated outlineprinted on the same media.

FIG. 9 is an illustration of a conventional print of a group sceneshowing six individuals with a reduced scale adjacent annotated outlineincorporating a legend and key printed on the same media.

FIG. 10a is an illustration of a conventional print of a group sceneshowing six individuals with an adjacent annotated outline reverseprinted on a section of transparency media including a fold line in the“open position”.

FIG. 10b is an illustration of a conventional print of a group sceneshowing six individuals with an adjacent annotated outline reverseprinted on a section of transparency media including a fold line in the“closed position” which overlays the conventional print section.

FIG. 11a is an illustration of a conventional print of a group sceneshowing six individuals with an alternative format adjacent annotatedoutline reverse printed on a section of transparency media including afold line in the “open position”.

FIG. 11b is an illustration of a conventional print of a group sceneshowing six individuals with an alternative format adjacent annotatedoutline reverse printed on a section of transparency media including afold line in the “closed position” which overlays the conventional printsection.

FIG. 12a is an illustration of group scene showing six individuals andannotated outline printed in a series of alternating horizontal linesincluding a lenticular or barrier overlay, in the first viewing positiondisplaying the group scene.

FIG. 12b is an illustration of group scene showing six individuals andannotated outline printed in a series of alternating horizontal linesincluding a lenticular or barrier overlay, in the second viewingposition displaying the annotated outline.

FIG. 13a is an illustration of group scene showing six individualsprinted on the front surface of a translucent display media andilluminated from the front side

FIG. 13b is an annotated outline of group scene showing six individualsreverse printed on the rear surface of a translucent display media.

FIG. 13c is an illustration of group scene showing six individualsprinted on the front surface of a translucent display media and anannotated outline of group scene showing six individuals reverse printedon the rear surface of a transflective display film illuminated from therear side.

DETAILED DESCRIPTION OF THE INVENTION

The present invention is particularly useful when used with imagesrecorded at formal gatherings featuring large numbers of individuals,such as academic class pictures or group photographs of members ofbusiness, social organizations, and sports teams. In these situations,individuals tend to be posed in rows facing the recording device and mayfeature hundreds of individuals, making ‘in scene’ printed information,annotation, keys, and legends impractical. The invention is also usefulfor photographs recorded at informal gatherings, where individuals tendto have more casual poses or is depicted engaged in activities such asrecreation, parties, picnics, reunions, and sporting events. Thephotographic prints produced from these events often have commemorativevalue to the participants and others associated with the individuals ororganizations. These hardcopy photo products can take the form ofprints, multi-image prints, duplex prints, panoramic prints, a printwith a transparent overlay, framed photographs, photo books, photo bookpages, brochures, yearbooks, class pictures, annual reports, posters,greeting cards, post card, wedding albums, and so forth.

FIG. 1 shows a graphic illustration of an image processing systemsuitable for practicing the image processing method and photo productproduction according to the present invention. The system may beembodied in a conventional personal computer, at a self-service imagingkiosk, on a touch tablet, or smart phone. The image processing requiredmay be performed on the computational device directly or via remoteprocessor connected to the local device by a communication network. Inaddition, the resulting hardcopy image product may be produced locallyor at a remote location. Imaging System 10 includes, Imaging Kiosk 20which is typically setup at retail locations for users to create andorder an array of photo products using digital images provided the user.Imaging Kiosk 20 includes; Touch Screen Display 30 providing a GraphicUser Interface (GUI) based user interaction, Media Read/Write Port Array40 for interfacing various forms of physical digital media, such asvarious formats of memory cards, USB flash drives, and for a wiredconnection to transfer images from mobile device 80.

Illustrated on Touch Screen Display 30 is Group Scene Image 100 providedin this example by the user's Wireless Mobile Device 80 via WirelessInteraction 70. Pointing Device 50 includes any suitable device thatprovides user interaction including; a computer mouse, track pad,drawing device and stylus, or camera, microphone, and speakercombinations for gesture and voice control. Also, included in imagingsystem 10 is optional Transaction Card Reader 60, to enable payment atretail venues and/or for use with customer loyalty cards to trackcustomer usage and to provide promotional incentives.

Imaging system 10 includes Wireless Interaction 70 for use with theuser's Wireless Mobile Device 80, and includes one or more wirelesscommunication techniques such as Bluetooth, WIFI, WIFI-Direct, and NearField Communications (NFC) that allow the user to communicate with theImaging System 10 and to transfer images from their mobile device 80, orvia their device in the case of images stored at a remote location, toImaging Kiosk 20. Additional devices for obtaining digital imagesprovided by the user include CD/DVD Reader/Writer 130 for still andmotion images stored on disks and Hard Copy Scanner 140 for images onprints or negative and transparency films. Examples of Hard Copy 140scanner include reflection and transparency scanners with manual platensto digitize hardcopy media, an automatic document feeder, and/or filmstrip and/or slide transport. The output devices for producing photoproducts include; Duplex Color Printer 90 which includes Thermaldye-sublimation, electro-photographic, inkjet, and/or Color Photo Paperprinter/processor type printers shown producing Group Scene Print 110.In addition, for printing large images for wall art, framing, or photoposters, Large Format Color Printer 120 is provided. Wireless Router 150provides network access to Imaging system 10 and its components. RemoteProduction Facility 160 is used to produce custom photo products, notsuitable for production at all retail venues, and include products suchas photo mugs, photo apparel, and high quality bound photo books.

Referring to FIG. 2a is a flow chart showing one embodiment of the imageprocessing method according to the present invention. The identificationof the individuals within the scene is obtained from facial recognition,metadata related to the recorded scene image, from a user operating auser input device and/or from image metadata information relating tosubject identification and subject placement within a scene provided bya social network system, such as Facebook, Google+, and/or as recordedby mobile device operating systems such as Android. Incorporated hereinby reference are U.S. Pat. No. 7,324,693 METHOD OF HUMAN FIGURE CONTOUROUTLINING IN IMAGES, U.S. Pat. No. 8,280,164 PRODUCING OBJECT CUTOUTS INTOPICALLY RELATED IMAGES, “Stationary Features and Cat Detection”,Journal of Machine Learning Research—François Fleuret and Donald Geman,U.S. Pat. No. 6,697,502 IMAGE PROCESSING METHOD FOR DETECTING HUMANFIGURES IN A DIGITAL IMAGE.

Alternative approaches to identifying and extracting subjects in a sceneinclude using the well-known chroma-key capture system to automaticallyisolate the foreground subjects from the background scene. In additionto so called “blue screen” or “green screen” chroma-key systems,techniques described U.S. Pat. No. 5,914,748 METHOD AND APPARATUS FORGENERATING A COMPOSITE IMAGE USING THE DIFFERENCE OF TWO IMAGESincorporated herein by reference, include recording two images of thescene one with the foreground subjects in position and one without theforeground subjects present in the scene. The two images are digitallyprocessed to create a pseudo-chroma-key effect allowing the foregroundsubjects to be isolated from the background.

FIG. 2a depicts system flowchart 170 which can be practiced on a photokiosk, mobile phone, tablet, or personal computer. In step 180 a userprovides a digital image which preferably contains two or moreindividuals and can originate from a digital camera, phone, tablet, ascan of hard copy print or film, or stills images extracted from video.In step 190 the user selects Info Print Feature by using a Graphic UserInterface and touch screen, pointing device, voice command, or any otherwell-known computer/user interaction technique, not shown. Next indecision Step 200 the system interrogates the metadata associated withthe digital image provided in step 180 to determine if the identitiesand locations for the individuals depicted in the digital image sceneare known. The associated metadata related to the identities andlocations within the scene of the depicted individuals can be obtainedfrom the digital image file, in a metadata database associated with thedigital image, from operating system such as Android, and/or from asocial network with a digital image capability. If at decision Step 200the identities and locations of the depicted individuals are known, theprocess continues to step 230 where the identities are associated withthe subjects depicted in the digital image in the scene. If at decisionStep 200 the identities and locations of the depicted individuals arenot known the process continues at decision Step 210 where the systemdetermines if the identities and locations of the depicted individualscan be determined. The system uses face identification techniques todetermine the relative positions of the depicted individuals and facerecognition techniques to identify the depicted individuals. If atdecision Step 210, the system fails to determine any or all theidentities and locations of the depicted individuals in group sceneImage 100 the process continues to graphic user interface step 220 wherethe imaging kiosk 20 requests identities and locations of the subjectsdepicted on touch screen display 30 from the user. The user would thenuse touch screen display 30 and/or pointing device 50 to provide therequested information to imaging kiosk 20. The user provided identitiesand locations of the individuals in group scene image 100 depicted ontouch screen 30 are then stored in association with the user provideddigital image at Step 230.

Now that the identities and locations of the subjects in group sceneimage 100 are stored in association with the user provided digital imagethe user selects the presentation format at Step 240 for the hardcopyphoto product to be created. The various presentation formats for thehardcopy photo products all provide the identity information and avisual reference relative to the original scene. The annotated visualreference may be printed on the same side of the same media and adjacentto the original scene, on the back side of the same media as theoriginal scene, in the same or difference scale, on a separateconventional or transparent media used in conjunction with the originalscene print, on a multiple image viewing print media incorporating abarrier or lenticular overlay, or on a duplex translucent media withscene image on one side and an aligned and reverse printed annotatedvisual scene derived reference on the other side of the translucentmedia. The details of the presentation formats of the hardcopy photoproducts will be discussed in detail later.

At Step 250, human faces and bodies are detected, the details of whichwill be addressed in the descriptions of FIG. 2b . Once the faces andbodies are detected the image is segmented in Step 260 and in Step 270,the human face and body segments are identified. In Step 280 theidentified segments are associated with each individual subject in thescene. A contiguous line is created around the human face and bodysegments associated with each individual subject in Step 290. Thedigitally created line can be produced in any color to contrast with thescene or subject image and can have a fixed or variable width that canbe selected by the user or be calculated to scale to selectedpresentation format, photo product type, image size, and/or imageresolution. In addition, the lines created around the individualsubjects can be continuous, hashed, dotted and so forth and individualline types, widths, and colors can also be selected for each individualsubject depicted in the scene. Furthermore, when individual line typesare used to depict different individuals, the line types can also bereferenced in the visual legend. At Step 300 the Z-axis depthinformation between the various individuals in the scene is maintainedso that outlined portions of the individuals appear to overlap accordingto their original positions relative to the scene image capture device.This is particularly useful for large group scenes with two or more rowsof individuals. More details of this process will be provided with thedescription of FIG. 2c . In addition, if subject position and depthinformation has been recorded as metadata, it can be used to assistand/or verify the Z-axis determination of the individuals within thescene.

At Step 310, to further create the outline image, the subject imageinformation from inside of the contiguous lines around each individualis removed and Step 320 the background image information outside of thecontiguous lines is removed, leaving only the contiguous lines. Turningnow to Step 330, the subject identity information is added according tothe appropriate subject positions, identities, and selected presentationformat. Positioning and formatting the subject identity information,outline image, and the original scene image is accomplished by eitherfixed or dynamic templates that are selected or modified according tothe selected presentation format. Once the positioning of the individualcomponents and the formatting is completed, at Step 340 the info printis produced on a hard copy printer compatible with selectedpresentation, or alternatively the formatted image file can be stored ortransmitted for fulfillment at a later time. The formatted image filecan be transmitted to a local retailer for in store printing and pick upby the customer or a designated recipient.

Turning now to FIG. 2b , an outline flow chart 350 showing the imageprocess method for creating the silhouette outlines around theidentified subjects. To produce the outlines of the subjects foundwithin the picture, object segmentation is employed. Those skilled inthe art will readily recognize that there are many methods of objectsegmentation. Often these methods require prior knowledge of the objectto be segmented, assistance from a user, or a combination of both.

Graph cut methods have become popular for optimizing the location of acontour. Graph cut methods are able to find a global optimum for binarylabeling (i.e., two labels) problems, such as foreground/backgroundimage segmentation. One popular method was described by Y. Boykov andM-P. Jolly in, “Interactive Graph Cuts for Optimal Boundary & RegionSegmentation of Objects in N-D Images” Proceedings of “InternationalConference on Computer Vision”, Vancouver, Canada, July 2001 Volume I,p. 105. Several variations to this method are described by F. Yi and I.Moon in “Image Segmentation: A Survey of Graph-cut Methods,” 2012International Conference on Systems and Informatics (ICSAI 2012), YantaiChina. These papers are incorporated herein by reference.

Boykov and Jolly introduce an interactive method of graph-cutsegmentation in which the user sets hard constraints (often referred toas “seeds”) which declare some pixels of the image to depict foregroundsubject matter and others to depict background. These hard constraintsare typically very small subsets of the foreground and background pixelsrespectively. A graph cutting algorithm is employed to find an optimalcontour which separates foreground from background. Several variationsof this algorithm use assumptions about the spatial distribution of thesubject to minimize or eliminate the dependency on the user. Forexample, some methods assume that pixels contacting or near the edge ofthe image will not depict foreground matter and that the subject will becentrally located. For the present invention, face detection can beemployed to further refine the location of foreground seeds. Pixels fromthe head region as well as pixels from the mid-chest region may be useda foreground seeds.

Yi and Moon also describe graph-cut extensions which employ shape-basedprior knowledge of the foreground subject to guide the determination ofthe optimal cut. Y. Yang and D. Ramanan, describe a method of estimatinga human pose in “Articulated pose estimation with flexiblemixtures-of-parts,” Computer Vision and Pattern Recognition (CVPR), 2011IEEE Conference on. IEEE, 2011. This paper is incorporated herein byreference. The method described by Yang and Ramanan can produceestimates of the location of body parts. These estimates areinsufficient to produce contour outlines by themselves. Rather, they canbe used to provide shape-based prior information as described by Yi andMoon, as well as providing good estimates of pixels which can be used asforeground seeds. In working with an image containing several people,the contour for each person is created independently in the preferredembodiment. In a second step, contours from different people are orderedfrom back to front and rendered into the final image.

In FIG. 2b , an image is provided to the system in get image Step 360.Faces are detected in detect faces Step 370. A face is selected selectface Step 380. The set expected boundaries Step 390 are determined forthe face. These boundaries are set such that given the head size, poseand location, the body of the person should be fully enclosed by theboundary. Optionally, the articulated pose of the body may be detectedusing the method described by Yang and Ramanan or other method todetect/estimate pose Step 400. If not, a prior probability estimate ofbody extent may be used. The outer boundary may be used to create theterms relating to background region, pixels with the highest probabilityof head and body are used to set the foreground regional terms in theset regional terms Step 410. The color difference between neighboringpixels is used to create boundary terms and set boundary terms in Step420. Shape terms are created using the information from set shape termsStep 430. A segmentation is obtained using the graph-cut algorithm incut graph Step 440. The contour is created from the boundary of thelargest region of the results of the create contour Step 450. Theprocess is repeated at the find more faces decision Step 460 for eachface at the select face Step 380. Finally, each of the contours ismerged in the merge contour results Step 470, which will be detailedbelow.

FIG. 2c illustrates flow chart 480, the method of merging contours. Theset of contours from an image are provided in Step 490. For eachcontour, the face detection information is provided Step 500. Thisinformation includes the detected face size, the detection location, therelative pose of the face and the estimated or actual age and gender ofthe person depicted. This information is used to estimate the relativedistance from the camera of each subject in Step 510. This relativedistance is referred to as Z-order. The set of contours are sorted bythe Z-order determined in Step 520. Finally, the contours are combinedby rendering them from farthest to nearest to the camera in the rendercontours Step 530.

Turning now to FIG. 3a , an illustration of a soft display of a groupscene 540 including group of six individuals 550. The soft display groupscene can be displayed on touch screen display 30, of imaging kiosk 20and/or on the screen of mobile device 80. FIG. 3b illustrates the softdisplay of a group scene 540, including the group of six individuals550, and with an outline constructed around each individual 560. In FIG.3c the illustration of the soft display of a scene 540 including a groupscene showing six individuals with an outline constructed around eachindividual 560, and the interior area of each individual of the group ofsix individuals is erased 570. FIG. 3d is an illustration of a softdisplay of a scene including a group of individuals 540, an outlineconstructed around each individual 560 of the group of six individualswhich have been erased 570. In addition the soft copy display scenebackground has been erased 580, with the only image informationremaining outline constructed around each individual 560. This techniqueprovides an easy to comprehend graphic visual reference that is alsouseful if the printed photo product includes a transparent overlay withthe legend information and outline graphic which can be used as neededproviding a temporary, minimally obscuring visual reference. The scenederived individual subject outlines appear to be “stacked” relative tothe subject's appearance in the original scene, with subjects in theforeground or front row appear on top of or in front of the subjects inthe background or back row appear below or in back of the foregroundsubjects. This is done to preserve the physical context of the originalscene by providing the Z-axis positions of the individuals relative tothe position of the camera recording the original scene.

FIG. 4a is an illustration of a front surface of a duplex printdepicting a scene including a group of six individuals 590. Optionally,multiple images can be printed on the front surface of the page, as in aphotobook, collage, or calendar prints. FIG. 4b illustrates the rearsurface of the duplex print shown in FIG. 4a . Arrow symbol 600indicates that the print has been turned over depicting the rear surfaceof a print of a scene including the outlines of a group of sixindividuals 610. In this embodiment, the group scene showing theoutlines of a group of six individuals with textual annotations 630,such as “Candice Thompson” identifying each individual depicted. Therear surface of the print 610 is used to provide a visual reference toidentity the individuals on the front surface of the print 590 accordingto the present invention. The rear surface of the print 610, as with thefront surface 590 can include multiple visual legends, identifiers, orcaptions containing automatically generated outlines depending on thenumber of scene images are depicted on the front surface.

FIG. 5 is an illustration that depicts an alternative embodiment of thepresent invention. Hardcopy print 640 is produced on either a duplex orsingle sided printer, since in this particular embodiment only the frontsurface for the hardcopy print contains images and graphic indicators.Hardcopy print 640 includes a section 650 of the front surface of theprint that includes a scene of a group of six individuals 550. Anillustration depicting the same group scene showing six individuals inoutline 560 is in another section 660 of the front surface of the print640. Included with the each of the outlines of a group of sixindividuals 560 is an alpha numeric indicator of the identity of eachone of the individuals 670. In a third section of the front surface ofthe print 640 is a legend including the keys and identities of theindividuals 680. Legend 680 includes a key matching each of the alphanumeric indicators 670 and information indicating the identity of theindividuals 690 depicted in outline 560.

FIG. 6a is an illustration of the front surface of a print conventionalhardcopy print of a group scene including a group of six individuals590. FIG. 6b is an illustration of a transparency print 700 of a sceneincluding the outlines of a group of six individuals 620. Each group ofsix individuals includes identification label 630 within the outlines ofeach individual. FIG. 6c is an illustration of a transparency print 700aligned and overlaying the conventional print 710 of a group sceneshowing six individuals 590. In this embodiment, the user can place thetransparency print 700 over the conventional print as need to identifythe subjects depicted in the scene.

FIG. 7 is an illustration of the front surface conventional print 720. Asection on the front surface 650 of the print 720 includes scene a groupof six individuals 550. Adjacent to section 650 is a second section 730including an annotated outline printed on the same surface of the samemedia. The section 730 of the print includes the outlines of the groupof six individuals 620, and within each outline there is an annotationidentifying and individual 630. This alternative embodiment is suitablefor hardcopy output and/or the printed image is large enough toaccommodate the annotation information within the individual outlines.

FIG. 8 is an illustration of the front surface conventional print 740. Asection on the front surface 650 of the print 740 includes scene a groupof six individuals 550. Adjacent to section 650 is a second section 750including an outline 620 printed on the same surface of the same media.The section 750 of the print includes the outlines of the group of sixindividuals 620 and external to each outline there is an annotation witha directional indicator identifying an individual 760. This alternativeembodiment is suitable for hardcopy output and/or the printed image hasempty background space large enough to accommodate the annotationinformation outside of the individual outlines.

FIG. 9 is an illustration of the front surface of a conventional print770, including a section of the print 650 including scene a group of sixindividuals 550. A second section 780 includes a reduced scale outlineof the group of six individuals 560. In addition, within each outline isan alpha numeric indicator 670 for use in conjunction with legend 680located on a third section of the front surface of print 770. In a 680adjacent to the reduced scale outline 560 of the front surface ofconventional print 770 includes a legend that provides the keyscorresponding to the alpha numeric indicator 670 and identities 690 ofthe individuals 560 depicted in section 650. Section 780 is shown inreduced scale, but may be provided in a different reduced or optionallyenlarged scale depending on the hardcopy print size, format, or scenecontent.

FIG. 10a is an illustration of a hardcopy print 790 with aconventionally printed reflective portion 840 and reverse printedtransparent portion 810 shown in the “open position”. Included in theillustration is a symbol indicating a fold line 800 between thereflective print portion of a print media and the transparent portion ofthe print media. There are several approaches to providing a hardcopyprint with a reflective and transparent portion, including; using atransparent media used in conjunction with a printer capable of printinga white reflective base layer on a portion of a transparent media orusing a pre or post printing method of assembling a reflective andtransparent media into a singular media piece. Additionally, sheet orroll transparent media can be pre-treated with a reflective, preferablywhite, printable surface on a portion approximating half of the surfaceof the sheet or roll transparency media. The printable surface can beapplied by printing, coating, or lamination and configured to becompatible with the intended printing technique. These pre-assembledconventional print and transparency media sheets or rolls can be usedwith any suitable digital inkjet, dye sublimation, electrophotographic,and or AgX printer. The transparency media can be either bound into thebook assembly or provided as an insert such as a book mark. Postprinting assembly using suitable pressure sensitive, thermal, orchemical adhesives or mechanical binding techniques and assemblies suchas staples, stitching, clips or the like. Alternatively, a transparencymedia alone can be used in conjunction with a printer capable ofprinting a reflective base layer, such as white, to a portion of thetransparent media to produce a reflective print portion.

The left side reflective portion 840 of print 790 includes the groupscene of six individuals and on the right side the transparency section810 includes a reverse printed annotated outline of the group of sixindividuals 820 and reverse printed annotations of the group of sixindividuals 830. Portions 840 and 810 are separated by fold line 800illustrated by a dashed line. In addition, Symbol 850 indicatesdirection in which the print 790 is intended to be folded at the foldline 800.

Alternative embodiments of applications this technique include oppositepages in a book, pamphlet, or card with the scene image on one side andthe transparency media reverse printed annotated outline printed on theadjacent page. The transparency can be attached to an edge of theconventional print image and folded into the overlay position whenrequired or can be unattached and of the same relative size and formatand placed over the conventional image as needed. In addition, the twoportions, 810 and 840 of print 790 can be bound with a permanent ortemporary adhesive or mechanical clips that can be removed.

FIG. 10b is an illustration of a print 790 of a group scene showing sixindividuals 840 with an adjacent annotated outline reverse printed on asection of transparency media 810 and the fold operation shown byindicator 850 along fold line 800 to place print 790 into the “closedposition” 860. In closed position 860, group scene reflective print 840is now overlaid by transparency portion 810 of print 790 and the of thereverse printed annotated outline, no longer appears to be reverseprinted and aligns with the reflective print group scene 840.

FIG. 11a is an illustration of an alternative to the embodiment shown inFIG. 10a and FIG. 10b including similar materials and techniques. Aprint of a scene including the group of six individuals on the bottom ofa reflective print portion 910 of a print media 870 and a reverseprinted and inverted annotated outlines of the group of six individuals880 on the top of a transparent portion of the print media 870. Thereverse printed and inverted annotated outlines of the group of sixindividuals 890 includes reverse printed and inverted annotations of thegroup of six individuals 900. A symbol indicating a horizontal fold lineand direction 800 to fold print media 870 from the “open position” shownin FIG. 11a to the “closed position” shown in FIG. 11 b.

FIG. 11b is an illustration the embodiment shown in FIG. 11a now in the“closed position” 920. The print 870 has been folded along the fold lineand direction 800 shown in FIG. 11a . The transparent portion 880 is nowaligned and overlaid on top of reflective print 910.

FIG. 12a is an illustration of a multi-view print in the first viewingposition 930 displaying the group scene showing six individuals 950 andincorporating a lenticular or barrier overlay 940. Not shown in FIG. 12ais that group scene 950 printed in a series of alternating horizontallines with outline 890 and annotations 990, shown in FIG. 12B, overlaidwith lenticular or barrier overlay 940. Multi-view prints are well knowndigital printing techniques used for producing sequence of stills imagesfor “motion prints” or a set of individual images used “multi-viewprints”, each of which has the ability to display a different image asthe print is viewed from a different angle. The technique requires alenticular clear plastic cover sheet consisting of a series ofhorizontal lenticules or a barrier overlay layer of opaque material witha series of horizontal precision slits 940. Either of these overlays isused with a processed digital image that is created with interspersedalternating horizontal lines obtained from the original video sequenceor set of still images. As the user holds the finished motion ormulti-view print and tilts the print, as illustration by directionalsymbol 960, relative to her viewing angle, the image that appearschanges showing a short motion clip or individual still images of theset of images. These techniques are described in detail in the U.S. Pat.No. 6,198,544 METHOD AND SYSTEM FOR CAPTURING IMAGES FROM A VIDEO TAPEFOR THE PURPOSE OF CREATING A MOTION CARD.

FIG. 12b is an illustration of group scene showing six individuals andannotated outline printed in a series of alternating horizontal linesincluding a lenticular or barrier overlay, in the second viewingposition displaying the annotated outline. A multi-view printincorporating a lenticular or barrier in the second viewing position970, a printed image of the annotated outlines of the group of sixindividuals, printed in a series of equally spaced and separatedhorizontal lines 980, the annotation identifying one of the individuals990. An alternative embodiment of this technique, not shown, includesincorporating a third set of equally spaced horizontal image segmentsincluding group scene 950, outlines 980, and annotations 990, so thatthree viewing angle dependent images are available. This would providean additional image that would incorporate the group scene image 950,outlines 980, and annotations 990 all simultaneously visible at one ofthe viewing positions.

FIG. 13a is an illustration of group scene showing six individuals 1010printed on the front surface of a translucent display media 1000. Asuitable translucent media capable of duplex printing, such as “KODAKPROFESSIONAL UV-Curable Display Film—Plus” which can be back-lit orfront-lit can be used for this application. Any suitable translucentprint media capable of accepting inkjet, silver halide,electrophotographic, thermal, or other duplex digital image printingtechniques may be used for this embodiment. Alternatively, two pieces ofsingle sided translucent media can be used if it assembled “back toback” after printing or transparent media can also be used if a whitelight diffusion layer is added in between two sheets of printedtransparent media. When single sided media is used it can be attachedmechanically with a frame, clips, or staples, or laminated using heat,chemical, of pressure sensitive adhesives. As taught in U.S. Pat. No.2,225,307 CONVERTIBLE PICTURE, one printed matter on one side of atransparent, or translucent, material, and another matter on theopposite side of the same material, to appear, or be visible bydifferent, or suitable light effects. FIG. 13a shows the image printedon the front surface of a translucent print 1000 of the scene depicting6 individuals 1010, being illuminated from the front side as indicatedby symbol 1020. Since only the front surface of the translucent print isbeing illuminated, only the front image of the group scene 1010 isvisible. In a normal usage scenario, a user would view with conventionallighting overhead which would typically illuminate the front surface ofthe print. A symbol indicating the translucent print has been turnedexposing the rear surface 1030, the results which will be shown in FIG.13 b.

FIG. 13b is a reverse printed image annotated outline of the group of 6individuals 1050 and reverse printed annotations of the group of 6individuals 1060 on the rear surface of a translucent display media1040. The illustration shows rear surface of the translucent print inorder to convey that the outline 1050 is aligned with the group scene1010 shown in FIG. 13a . A symbol indicating the translucent print hasbeen turned exposing the front surface 1070, the results which will beshown in FIG. 13 c.

FIG. 13c is an illustration of a symbol indicating that the translucentprint is being illumination from the rear 1080, the rear illuminatedtranslucent print depicting a group scene of 6 individuals and annotatedoutline of the group of 6 individuals identifying the individuals. Inthis usage scenario, the user would typically hold the print up toexpose the rear surface of the print to overhead lighting or held infront of a lamp or bright window to make the annotated outline printedon the rear surface of the print visible while viewing the front surfaceof the print.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

GLOSSARY OF DESCRIPTIVE TERMS

“Annotated Visual Scene Derived Reference”—an annotated line graphicautomatically created from the original group scene. The VisualReference includes outlines of the individuals within the scenemaintaining their original depth information relative to the scenerecording device (camera) and providing identification annotation inassociation the outlines of the individuals.

“Annotated Outline”—a visual graphic outline surrounding individualsubjects within a clustered group in order to visually identifyindividuals within the group, including maintaining the “Z-axis” or rowpositions of the outlines so that they overlap properly to visuallyconvey the positions of the individuals depicted in the original groupscene. By maintaining only the outlines of individuals depicted in thescene, an easy and simple visual reference is created showing therelative proximity of individuals in group shots or scenes with multipleindividuals.

“Individual subject”—a person or animal, such as a family pet, with anindividual name designation, as opposed to a generically recognizedsubject. For example, “Bob Jones” or “Skippy” are identified individualsubjects as opposed to a “Human Male” or “Dog”, which would beintuitively obvious but the individual's identity would remain unknownto the viewer without prior knowledge of the individual.

“Non-obscuring”—printed media means for providing the information andvisual references without interfering with the visual scene as with avisual reference printed adjacent to the visual scene or the option toselect and remove the visual reference as with an overlay, multi-viewprint, or duplex printed translucent material.

“Group Scene”—two or more individuals, including human and animals, withknown or discoverable individual identities in a formal pose orspontaneous casual setting within the same field of view of therecording device (camera).

“Automatically creating a line graphic”—using one or more objectrecognition and segmentation algorithms to identify human and animalsubjects in a scene and to produce an outline around each individualsubject in a stacked arrangement relative to the position of theoriginal scene recording device (camera).

PARTS LIST

-   10 Imaging System-   20 Imaging Kiosk-   30 Touch Screen Display-   40 Media Read/Write Port Array-   50 Pointing Device-   60 Transaction Card Reader-   70 Wireless Interaction-   80 Wireless Mobile Device-   90 Duplex Color Printer-   100 Group Scene Image-   110 Group Scene Print-   120 Large Format Color Printer-   130 CD/DVD Reader/Writer-   140 Hard Copy Scanner-   150 Wireless Router-   160 Remote Production Facility-   170 System Flowchart-   180 “Provide Image” Step-   190 “Select Info Print Feature” Step-   200 “Identities/Locations Known?” Decision Step-   210 “Can Identities/Locations be Determined?” Decision Step-   220 “Request Identities/Locations GUI” Step-   230 “Associate Subject Identities” Step-   240 “Select Format” Step-   250 “Detect Faces/Bodies” Step-   260 “Segment Images” Step-   270 “Identify Segments” Step-   280 “Associate Segments” Step-   290 “Create Line” Step-   300 “Maintain Z-Order” Step-   310 “Remove Subject Image” Step-   320 “Remove Background Image” Step-   330 “Add Subject identity Info” Step-   340 “Make Info Print” Step-   350 Outline Flow Chart-   360 “Get Image” Step-   370 “Detect Faces” Step-   380 “Select Face” Step-   390 “Set Expected Boundaries” Step-   400 “Detect/Estimate Pose” Step-   410 “Set Regional Terms” Step-   420 “Set Boundary Terms” Step-   430 “Set Shape Terms” Step-   440 “Cut Graph” Step-   450 “Create Contour” Step-   460 “More Faces” Decision Step-   470 “Merge Contour Results” Step-   480 Z-Order Position Flow Chart-   490 “Get Contours” Step-   500 “Get Face Detection Info” Step-   510 “Determine Z-order of faces” Step-   520 “Sort Contours based on Z-order” Step-   530 “Render Contours” Step-   540 Soft display of a scene including a group of individuals-   550 Group of six individuals-   560 An outline constructed around each individual-   570 Group of six individuals erased-   580 Scene back ground erased-   590 Front surface of a print of a scene including a group of six    individuals-   600 Symbol indicating the print has been turned over-   610 Rear surface of a print of a scene including the outlines of a    group of six individuals-   620 Outlines of a group of six individuals-   630 Annotation identifying and individual-   640 A print of a scene a group of six individuals and the outlines    of a group of six individuals-   650 A section of the print including scene a group of six    individuals-   660 A section of the print including the outlines of a group of six    individuals-   670 Key indicating the identity of one of the individuals-   680 A legend including the keys and identities of the individuals-   690 A key and identity of one of the individuals-   700 A transparency print of a scene including the annotated outlines    of a group of six individuals-   710 The transparency print of a scene overlaying the print of the    scene including a group of six individuals-   720 A print of a scene including the group of six individuals and    the annotated outlines of the group of six individuals-   730 A section of the print including the annotated outlines of a    group of six individuals-   740 A print of a scene including the group of six individuals and    the outlines of the group of six individuals-   750 A section of the print including annotated indicators associated    with outlines of a group of six individuals-   760 An annotated indicator indicating the identity of one of the    individuals-   770 A print of a scene including the group of six individuals and    the reduced scale outlines of the group of six individuals-   780 A section of the print including the reduced scale outlines of a    group of six individuals-   790 A print of a scene including the group of six individuals on the    right side of a reflective print portion of a print media and the    reverse printed annotated outlines of the group of six individuals    on the left side of a transparent portion of the print media-   800 A symbol indicating a fold line between the reflective print    portion of a print media and the transparent portion of the print    media-   810 The reverse printed annotated outlines of the group of six    individuals on the left side of the transparent portion of the print    media-   820 The reverse printed annotated outlines of the group of six    individuals-   830 The reverse printed annotations of the group of six individuals-   840 The print of a scene including the group of six individuals on    the left side of a reflective print portion of a print media-   850 Symbol indicating the print has been folded at the fold line-   860 The transparency portion of the print of the reverse printed    annotated outline folded and overlaying the print of the scene    including a group of six individuals-   870 A print of a scene including the group of six individuals on the    bottom of a reflective print portion of a print media and the    reverse printed and inverted annotated outlines of the group of six    individuals on the top of a transparent portion of the print media-   880 The reverse printed and inverted annotated outlines of the group    of six individuals on the top transparent portion of the print media-   890 The reverse printed and inverted annotated outlines of the group    of six individuals-   900 The reverse printed and inverted annotations of the group of six    individuals-   910 The print of a scene including the group of six individuals on    the bottom of a reflective print portion of a print media-   920 The transparency portion of the print of the reverse printed and    inverted annotated outline folded and overlaying the print of the    scene including a group of six individuals-   930 A multi-view print incorporating a lenticular or barrier in the    first viewing position-   940 The lenticular or barrier cover sheet-   950 A printed image of the scene depicting 6 individuals, printed in    a series of equally spaced and separated horizontal lines-   960 Symbol indicating the changeable viewing position of the    multi-view print-   970 A multi-view print incorporating a lenticular or barrier in the    second viewing position-   980 A printed image of the annotated outlines of the group of six    individuals, printed in a series of equally spaced and separated    horizontal lines-   990 The annotation identifying one of the individuals-   1000 A translucent print of the scene depicting 6 individuals, with    the image printed on the front surface of the print-   1010 A scene depicting 6 individuals-   1020 A symbol indicating that the translucent print is being    illumination from the front-   1030 A symbol indicating the translucent print has been turned    exposing the rear surface-   1040 The rear surface of the translucent print depicting the reverse    printed annotated outline of the group of 6 individuals-   1050 Reverse printed image annotated outline of the group of 6    individuals-   1060 Reverse printed annotations of the group of 6 individuals-   1070 A symbol indicating the translucent print has been turned    exposing the front surface-   1080 A symbol indicating that the translucent print is being    illumination from the rear-   1090 A rear illuminated translucent print depicting a group scene of    6 individuals and annotated outline of the group of 6 individuals    identifying the individuals

1. A method of creating a hardcopy photo product of a recorded scene ofa group of individuals, the method comprising: a) obtaining, at aprocessor, an image of a scene with two or more individuals; b) theprocessor analyzing the scene to automatically create a line graphicthat outlines the individuals within the scene while maintaining theirpositions relative to the scene recording device; c) the processorextracting the created line graphic image from the scene image; d) theprocessor obtaining identification of individuals within the scene; e)the processor annotating the created line graphic image with theobtained identification information; f) the processor formatting therecorded scene image and the annotated line graphic image to produce ahardcopy photo product that includes the annotated line graphic image toprovide visual identification information related to the individualsdepicted in the scene image without obscuring the scene image; and e)producing the formatted hardcopy photo product.
 2. The method of claim 1wherein the identification of the individuals within the scene isobtained from facial recognition.
 3. The method of claim 1 wherein theidentification of the individuals within the scene is obtained frommetadata related to the recorded scene image.
 4. The method of claim 1wherein the identification of the individuals within the scene isobtained from a user operating a user input device.
 5. The method ofclaim 1 wherein the identification of the individuals within the sceneis obtained from information provided by a social network.
 6. The methodof claim 1 wherein the two or more individuals within the scenecomprises people and/or animals, with individual name designations. 7.The method of claim 1 wherein the method of the invention is performedon a phone, tablet, personal computer, or retail photo kiosk eitherlocally on the device or connected via a communications network to aremote processor.
 8. The method of claim 1 wherein the formatting of thephoto product includes the scene image of one side of a print media andthe annotated line graphic image on the other, the scene image and theannotated line graphic image on separate portions of the same side of aprint media, the scene image printed on reflective media and theannotated line graphic image printed on a separate transparent printmedia, the scene image and a reduced scale version of the annotated linegraphic image on separate portions of the same side of a print media,the scene image printed on reflective portion of a print media and theannotated line graphic image is reverse printed on an adjacent andattached transparent print media, the scene image and the annotated linegraphic image are interspersed and printed on a lenticular or barriermulti-view print media, or the scene image printed on front surfacetranslucent or transparent print media and the annotated line graphicimage is reverse printed and aligned to the scene image on the rearsurface of the translucent or transparent print media.
 9. The method ofclaim 1 wherein the format for the photo product is selected from agroup of softcopy displayed format options by the user.
 10. The methodof claim 1 wherein the analysis of the scene content to automaticallycreate a line graphic that outlines the individuals within the scenemaintaining their positions relative to the scene recording deviceincludes at least one algorithmic technique including graph cut imagesegmentation, edge detection, pose consistency, pose clustering,scale-invariant feature transform (SIFT), speeded up robust features(SURF), and/or simple interactive object extraction (SIOX).
 11. Themethod of claim 1 wherein the processor extracting the created linegraphic image from the scene image comprises the processor automaticallyextracting the created line graphic image from the scene image.
 12. Themethod of claim 1 wherein the processor obtaining identification ofindividuals within the scene comprises the processor automaticallyobtaining identification of individuals within the scene.
 13. The methodof claim 1 wherein the processor annotating the created line graphicimage with the obtained identification information comprises theprocessor automatically annotating the created line graphic image withthe obtained identification information.
 14. The method of claim 1wherein the processor formatting the recorded scene image and theannotated line graphic image to produce a hardcopy photo product thatincludes the annotated line graphic image to provide visualidentification information related to the individuals depicted in thescene image without obscuring the scene image comprises the processorautomatically formatting the recorded scene image and the annotated linegraphic image to produce a hardcopy photo product that includes theannotated line graphic image to provide visual identificationinformation related to the individuals depicted in the scene imagewithout obscuring the scene image.
 15. The method of claim 1 furthercomprising automatically generating the hardcopy photo product with avisual annotated alpha numeric graphic legend to provide additionalinformation related to the printed image without obscuring the printedimage.